Automatic Tracking Mode For Controlling An Unmanned Aerial Vehicle

ABSTRACT

Some embodiments include methods performed by a processor associated with a wireless communication device for enabling an unmanned autonomous vehicle (UAV) to operate in an automatic user tracking mode. Such embodiments may include capturing image data of surroundings by a camera while the UAV is operating in the automatic user tracking mode, calculating estimated position information for the wireless communication device based on captured image data, and transmitting estimated position information to the UAV for use in tracking a user of the wireless communication device. Some embodiments include methods performed by a processor of a UAV for enabling the UAV to automatically follow a user. Such embodiments may include calculating a current position of the UAV, receiving from a user&#39;s wireless communication device estimated position information derived from image data captured by a camera of the wireless communication device, and determining whether an update to the UAV motion is required.

BACKGROUND

Aerial vehicles such as unmanned aerial vehicles (UAVs) can be used forperforming surveillance, reconnaissance, and exploration tasks formilitary and civilian applications. Such aerial vehicles may carry apayload configured to perform a specific function, such as personalphotography and videography.

Conventional UAVs may be manually controlled by an operator via a remotecontrol device, requiring a dedicated operator of the UAV different fromthe subject of the photography or videography.

In some instances, it may be desirable for UAVs to track a specifictarget. For small-sized UAVs, such tracking is traditionally achievedvia control commands from a user-operated remote control terminal ordevice. Such manual tracking control may become difficult in certaincircumstances, such as when the target is moving quickly or is at leastpartially blocked from view of the user. Furthermore, the attentionnecessary for such manual tracking typically requires a dedicated userthat controls a camera that onboard the UAV separate from the user thatcontrols the navigation, thereby increasing the cost for UAV photographyand other applications.

SUMMARY

Various embodiments include methods performed by a processor associatedwith a wireless communication device for enabling an unmanned autonomousvehicle (UAV) to operate in an automatic user tracking mode. Variousembodiments may include capturing image data of surroundings by at leastone camera associated with the wireless communication device while theUAV is operating in the automatic user tracking mode, calculatingestimated position information for the wireless communication devicebased on the captured image data, and transmitting the estimatedposition information to the UAV for use in tracking the target user.

Some embodiments may further include detecting that the automatic usertracking mode is initiated based on detecting user inputs on thewireless communication device. Some embodiments may further includedetermining whether the UAV is still operating in the automatic usertracking mode, and repeating collecting image data, calculatingestimated position information, and transmitting the estimated positioninformation to the UAV in response to determining that the UAV is stilloperating in the automatic user tracking mode.

In some embodiments, calculating estimated position information mayinclude calculating a change in position of the wireless communicationdevice, and calculating the change in position may include analyzing asequence of the captured image data over a period of time to calculate arate of movement of at least one feature in the surroundings of a targetuser.

Some embodiments may further include obtaining updated location datareceived through a Global Positioning System (GPS) receiver associatedwith the wireless communication device, and transmitting the updatedlocation data to the UAV for use in tracking the target user. Someembodiments may further include obtaining inertial sensor output datafrom at least one of an accelerometer, a gyroscope, and a magnetometerassociated with the wireless communication device, and generatingcombined position information for the wireless communication devicebased on the estimated position information and the inertial sensoroutput data, in which transmitting the estimated position information tothe UAV may include transmitting the combined position information.

Some embodiments may further include transmitting initial targetinformation to the UAV, wherein the initial target information providesto the UAV an initial location of the wireless communication device. Insuch embodiment, the initial target information may provide to the UAVidentity data for at least one of the wireless communication device andthe target user. In some embodiments, the estimated position informationtransmitted to the UAV may be configured to enable the UAV to determinewhether an update to the UAV motion is required.

Some embodiments may further include receiving user input on thewireless communication device, and transmitting to the UAV one or moreflight command derived from the user input, wherein the one or moreflight command is transmitted over a wireless communication link withthe estimated position information. In some embodiments, calculatingestimated position information may include calculating a currentestimated position of the wireless communication device.

Further embodiments include a wireless communication device including atransceiver and a processor coupled to the transceiver, and configuredto perform operations of the methods summarized above.

Various embodiments include methods performed by a processor of a UAVfor enabling the UAV to automatically follow a user includingcalculating a current position of the UAV, receiving estimated positioninformation from a wireless communication device associated with theuser, in which the estimated position information is derived from imagedata of surroundings of the user captured by at least one camera of thewireless communication device, and determining whether an update to theUAV motion is required. In some embodiments, receiving the estimatedposition information from the wireless communication device may includereceiving a change in position calculated by the wireless communicationdevice. In some embodiments, receiving the estimated positioninformation from the wireless communication device may include receivinga current estimated position of the wireless communication device, andthe method may further include calculating a change in position of thewireless communication device using the received current estimatedposition.

In some embodiments, determining whether an update to the UAV motion mayinclude comparing the estimated position information to previousposition information received from the wireless communication device,and detecting movement of the wireless communication device based on thecomparison. In some embodiments, determining whether an update to theUAV motion may include comparing the estimated position information to acurrent UAV position, and determining whether the UAV has maintained aselected offset from the wireless communication device, wherein theselected offset may include a preset elevation or ground distance.

Some embodiments may further include receiving via a wirelesscommunication one or more flight commands from the wirelesscommunication device, determining whether implementing the one or moreflight commands would cause the UAV to collide with an obstacle, andusing the estimated position information to override or modify the oneor more flight commands in response to determining that implementing theone or more flight commands would cause the UAV to collide with anobstacle.

Further embodiments include a UAV including a transceiver and aprocessor coupled to the transceiver, and configured to performoperations of the methods summarized above.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and constitutepart of this specification, illustrate exemplary embodiments of theclaims, and together with the general description given above and thedetailed description given below, serve to explain the features of theclaims.

FIG. 1 is a block diagram illustrating components of a typical UAVsystem suitable for use in the various embodiments.

FIG. 2 is a diagram illustrating a UAV flying at a given standofflocation relative to a user based on location and movement informationcommunicated by a user's wireless communication device that images itssurroundings.

FIG. 3 is a block diagram illustrating a wireless communication deviceconfigured to control a UAV according to various embodiments.

FIGS. 4A-4C are process flow diagrams illustrating methods forsupporting on a wireless communication device an automatic user trackingmode implemented by a UAV according to various embodiments.

FIG. 5 is a process flow diagram illustrating a method for operating anautomatic user tracking mode on a UAV according to various embodiments.

FIG. 6 is a component block diagram of a UAV suitable for use with thevarious embodiments.

FIG. 7 is a component diagram of an example wireless communicationdevice suitable for use with various embodiments.

DETAILED DESCRIPTION

Various embodiments will be described in detail with reference to theaccompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.References made to particular examples and implementations are forillustrative purposes, and are not intended to limit the scope of theclaims.

Various embodiments provide an improved automatic tracking mode used tocontrol motion of an unmanned aerial vehicle (UAV) based on a controldevice associated with a target user. In particular, the control deviceis configured with at least one camera and various sensors (e.g.,accelerometer, gyroscope, magnetometer, etc.), which are used tocalculate position information (e.g., an estimated position, change inposition information, or combined position information) for the controldevice. Position information may be based on data from at least onesensor on the control device, which may determine a relative position ofthe target in the local environment.

The control device may utilize captured image data of the surroundingenvironment to calculate position information (e.g., change in positioninformation) associated with the target user, such as motion and/orchange in direction of the control device over the time between images.In some embodiments, the change in position information may beintegrated with output data from one or more inertial sensor (e.g., anaccelerometer, gyroscope, etc.) to generate combined positioninformation.

In various embodiments, the control device may, after calculating orgenerating position information (e.g., an estimated position, change inposition information, or combined position information), send theinformation to the UAV in addition to or instead of the target's GPSlocation. That is, the control device may send the position information,or the position information with GPS location data, to the flightcontroller of the UAV, which updates/adjusts the motion of the UAVaccordingly. Specifically, the UAV may apply some or all of the receiveddata to flight controls to adjust the UAV's position to maintain apredefined position relative to the user (control device). Inparticular, the UAV may determine its own estimated position based ondata from at least one sensor on the UAV. In some embodiments, thecurrent UAV position may also be determined based on the UAV's relativeposition in the surrounding environment instead of or in addition to theUAV's GPS location. Based on local estimated position information of theuser (which may be frequently provided to the UAV by the controldevice), and the current estimated position of the UAV, the UAV maydetermine whether and how to adjust its movement to accurately track thetarget user.

The use of information derived from the camera and/or sensor of thecontrol device in various embodiments, either alone or in combinationwith data from the control device's GPS receiver, may allow for a moreresponsive following of the user compared to using only the controldevice's GPS data for guiding the UAV.

Various embodiments may be useful with any of a number of vehicles,examples of which include UAVs, unmanned autonomous land vehicles,unmanned autonomous watercraft, and autonomous spacecraft. A UAV may beautonomous (self-navigating), remotely controlled, server controlled,beacon controlled, or may be controlled by a combination of controlmethods. Various embodiments may be particularly useful for aerial UAVsdue to the high mobility, and increasing applications and numbers ofUAVs, as well as the presence of restricted airspace throughout theworld.

The terms “Global Positioning System” (GPS) and “Global NavigationSatellite System” (GNSS) are used interchangeably herein to refer to anyof a variety of satellite-aided navigation systems, such as GlobalPositioning System (GPS) deployed by the United States, GLObalNAvigation Satellite System (GLONASS) used by the Russian military, andGalileo for civilian use in the European Union, as well as terrestrialcommunication systems that augment satellite-based navigation signals orprovide independent navigation information.

An example UAV 100 illustrated in FIG. 1 is a “quad copter” having fourhorizontally configured rotary lift propellers 101 and motors fixed to aframe 105. The frame 105 may support a controller 110, landing skids andthe propulsion motors, power source (power unit 150) (e.g., battery),payload securing mechanism (payload securing unit 107), and othercomponents.

The UAV 100 may be provided with a control unit 110. The control unit110 may include a processor 120, communication resource(s) 130,sensor(s) 140, and a power unit 150. The processor 120 may be coupled toa memory unit 121 and a navigation unit 125. The processor 120 may beconfigured with processor-executable instructions to control flight andother operations the UAV 100, including operations of the variousembodiments. In some embodiments, the processor 120 may be coupled to apayload securing unit 107 and landing unit 155. The processor 120 may bepowered from a power unit 150, such as a battery. The processor 120 maybe configured with processor-executable instructions to control thecharging of the power unit 150, such as by executing a charging controlalgorithm using a charge control circuit. Alternatively or additionally,the power unit 150 may be configured to manage charging. The processor120 may be coupled to a motor system 123 that is configured to managethe motors that drive the rotors 101. The motor system 123 includes oneor more propeller drivers. Each of the propeller drivers includes amotor, a motor shaft, and a propeller.

Through control of the individual motors of the rotors 101, the UAV 100may be controlled in flight. In the processor 120, a navigation unit 125may collect data and determine the present position and orientation ofthe UAV 100, the appropriate course towards a destination, and/or thebest way to perform a particular function.

An avionics component 129 of the navigation unit 125 may be configuredto provide flight control-related information, such as altitude,attitude, airspeed, heading and similar information that may be used fornavigation purposes. The avionics component 129 may also provide dataregarding the orientation and accelerations of the UAV 100 that may beused in navigation calculations. In some embodiments, the informationgenerated by the navigation unit 125, including the avionics component129, depends on the capabilities and types of sensor(s) 140 on the UAV100.

The control unit 110 may include at least one sensor 140 coupled to theprocessor 120, which can supply data to the navigation unit 125 and/orthe avionics unit 129. For example, sensors 140 may include inertialsensors, such as one or more accelerometers (providing motion sensingreadings), one or more gyroscopes (providing rotation sensing readings),one or more magnetometers (providing direction sensing), or anycombination thereof. Sensors 140 may also include GPS receivers,barometers, etc. Inertial sensors may provide navigational information,e.g., via dead reckoning, including at least one of the position,orientation, and velocity (e.g., direction and speed of movement) of theUAV 100. A GPS receiver may provide three-dimensional coordinateinformation of the UAV 100 via communication with one or more GPSsatellite. A barometer may provide ambient pressure readings used toapproximate elevation level (e.g., absolute elevation level) of the UAV100.

The control unit 110 may include at least one camera 127 and an imagingsystem 129. The imaging system 129 may be implemented as part of theprocessor 120, or may be implemented as a separate processor, such as anASIC, a FPGA, or other logical circuitry. For example, the imagingsystem 129 may be implemented as a set of executable instructions storedin the memory device 121 that execute on a processor 120 coupled to theat least one camera 127. Each of the cameras 127 may includesub-components other than image capturing sensors, includingauto-focusing circuitry, ISO adjustment circuitry, and shutter speedadjustment circuitry, etc.

The control unit 110 may include communication resource(s) 130, whichmay be coupled to at least one antenna 131 and include one or moretransceiver. The transceiver(s) may include any of modulators,de-modulators, encoders, decoders, encryption modules, decryptionmodules, amplifiers, and filters. The communication resource(s) 130 mayreceive control instructions (e.g., navigational mode toggling,trajectory instructions, general settings, etc.) from one or morewireless communication device 170.

In some embodiments, the communication resource(s) 130 may include a GPSreceiver, enabling Global Navigation Satellite System (GNSS) signals tobe provided to the navigation unit 125. Alternatively or in addition,the communication resource(s) 130 may include one or more radio receiverfor receiving navigation beacon or other signals from radio nodes, suchas navigation beacons (e.g., very high frequency (VHF) omnidirectionalrange (VOR) beacons), Wi-Fi access points, cellular network sites, radiostation, etc. In some embodiments, the navigation unit 125 of theprocessor 120 may be configured to receive information from a radioresource (e.g., 130). UAVs may navigate using navigation systems such asGNSS, Global Positioning System (GPS), etc. In some embodiments, the UAVmay use an alternate source of positioning signals (i.e., other thanGNSS, GPS, etc.). Because UAVs often fly at low altitudes (e.g., below400 feet), the UAV may scan for local radio signals (e.g., Wi-Fisignals, Bluetooth signals, Cellular signals, etc.) associated withtransmitters (e.g., beacons, Wi-Fi access points, Bluetooth beacons,small cells (e.g., picocells, femtocells, etc.), etc.) having knownlocations such as beacons or other signal sources within restricted orunrestricted areas near the flight path. The UAV 100 may use locationinformation associated with the source of the alternate signals togetherwith additional information (e.g., dead reckoning in combination withlast trusted GNSS/GPS location, dead reckoning in combination with aposition of the UAV takeoff zone, etc.) for positioning and navigationin some applications. Thus, the UAV 100 may navigate using a combinationof navigation techniques, including dead-reckoning, camera-basedrecognition of the land features below the UAV (e.g., recognizing aroad, landmarks, highway signage, etc.), etc. that may be used insteadof or in combination with GNSS/GPS location determination andtriangulation or trilateration based on known locations of detectedwireless access points.

The processor 120 and/or the navigation unit 125 may be configured tocommunicate with a wireless communication device 170 through a wirelessconnection (e.g., a cellular data network) via a communication resource(e.g., a radio frequency (RF) resource) 130 to receive assistance datafrom the server and to provide UAV position information and/or otherinformation to the server. The communication resource(s) 130 may includea radio configured to receive communication signals, navigation signals,signals from aviation navigation facilities, etc., and provide suchsignals to the processor 120 and/or the navigation unit 125 to assist inUAV navigation tasks.

The processor 120 may use a radio (e.g., 130) to conduct wirelesscommunications with one or more wireless communication device 170 suchas smartphone, tablet, or other device with which the UAV 100 may be incommunication. A bi-directional wireless communication link 132 may beestablished between transmit/receive antenna 131 of the communicationresource(s) 130 and transmit/receive antenna 171 of the wirelesscommunication device 170. For example, the wireless communication device170 may be a portable or wearable device of a user that the UAV isconfigured to track. In some embodiments, the wireless communicationdevice 170 and UAV 100 may communicate through an intermediatecommunication link such as one or more network nodes or othercommunication devices. For example, the wireless communication device170 may be connected to the UAV 100 through a cellular network basestation or cell tower. The wireless communication device 170 maycommunicate with the UAV 100 through local access node or through a dataconnection established in a cellular network.

In some embodiments, the communication resource(s) 130 may be configuredto switch between a cellular connection and a Wi-Fi connection dependingon the position and altitude of the UAV 100. For example, while inflight at an altitude designated for UAV traffic, the communicationresource(s) 130 may communicate with a cellular infrastructure in orderto maintain communications with the wireless communication 170. Anexample of a flight altitude for the UAV 100 may be at around 400 feetor less, such as may be designated by a government authority (e.g., FAA)for UAV flight traffic. At this altitude, it may be difficult toestablish communication with some of the wireless communication devices170 using short-range radio communication links (e.g., Wi-Fi).Therefore, communications with the wireless communication device 170 maybe established using cellular telephone networks while the UAV 100 is atflight altitude. Communication with the wireless communication device170 may transition to a short-range communication link (e.g., Wi-Fi orBluetooth) when the UAV 100 moves closer to the wireless communicationdevice 170.

While the various components of the control unit 110 are illustrated inFIG. 1 as separate components, some or all of the components (e.g., theprocessor 120, the motor control unit 123, the communication resource(s)130, and other units) may be integrated together in a single device orunit, such as a system-on-chip.

While conventional UAVs may be controlled to navigate according to usercommands and/or predetermined navigation paths, some UAVs areadditionally or alternatively configured to perform functions thatrequire automatically tracking a particular target. In conventionaltracking applications, target information may be used by the UAV tocause the imaging device to track the target so as to maintainpredetermined position and/or size of the target within one or moreimages captured by the imaging device.

Some UAV systems allow a single user to control both the navigation of aUAV and tracking of a target. For example, a user can employ a userinterface on a control device to specify the target to the UAV, whichthen tracks the target. Such tracking may be performed, for example,using an imaging device onboard the UAV. For example, the attitude,position, velocity, zoom, and other aspects of the UAV and/or theimaging device can be automatically adjusted to ensure that the usermaintains a designated position and/or size within the images capturedby the imaging device.

If the target is associated with a wireless communication device that isconfigured with one or more GPS receiver, the tracking may be performedbased on location information received from the device. The user mayalso change or adjust the target to track in real time using the controlterminal. In particular, current UAVs may be configured to also supportone or more tracking modes that enable the user and the target to be onein the same (e.g., a user tracking mode). Various user tracking modesmay include, for example, an autonomous user tracking mode (also called“Follow-me” mode) that automatically tracks the target user, and asemi-autonomous user tracking mode that can receive or incorporate userinput to override and/or supplement the automatic tracking. As withother tracking modes, the user tracking modes are typically controlledthrough the user's position within images captured by the UAV, and/orusing location information received from a device of the user configuredwith a GPS receiver.

By implementing a user tracking mode, the user is able to focus on otheractivities while being followed by the UAV in real time. In existingFollow-me modes, the control device may provide information about itsown location or movements to the UAV to help the UAV tracking. TheFollow-me mode may be applicable for security applications. For example,the UAVs may be instructed to follow a convoy where the control deviceis on-board a convoy, or walking around with in a variety of settings,including crowds.

However, the existing techniques employed in user tracking modes havelimitations that can reduce performance of the UAV during automatictracking. For example, when the control device provides GPS informationabout its own location, such information may be inaccurate orunavailable to a user who is in certain locations (e.g., indoors, etc.).GPS also experiences drift, and GPS receivers on control devicestypically have a low update rate (e.g., 1 Hz), causing the UAV to movein a delayed manner relative to the control device. Additionally, whenemploying line-of-sight tracking using an imaging device (e.g., camera),the UAV may lose track of the user, particularly when the UAV is a longdistance from the user.

Various embodiments improve automatic user tracking modes byimplementing enhanced estimated positions of the target user.Specifically, various embodiments may accurately and efficientlygenerate estimated position information using existing features on awireless communication device that is the target/control device for theUAV. Specifically, the control device in various embodiments may be anyof a variety of wireless communication devices worn by, carried by, orassociated with the user, such as a smartphone, tablet, wearablecomputing device, etc., and may be configured with a variety ofperipheral components, including sensors. In some embodiments, one ormultiple of these sensors and/or other components may be leveraged bythe control device to calculate estimated position information orenhance the existing calculation of estimated position information. Forexample, the control device may be configured, in addition to one ormore GPS receiver, with at least one camera and various inertial andother sensors (e.g., accelerometer, gyroscope, magnetometer, etc.). Thecontrol device may utilize data collected from these features forcalculating estimated position information for the target user, eitheralone or in conjunction with GPS data. In various embodiments, thecontrol device may calculate its estimated position by implementingvarious algorithms (e.g., sensor fusion algorithms) using data obtainedby one or more camera, and optionally from sensors and/or a GPS receiveron the control device. In various embodiments, calculating estimatedposition information for the control device may involve using one ormore sensor fusion algorithm to calculate the relative position of thecontrol device in its environment, the control device velocity, thecontrol device orientation, etc.

In various embodiments, the control device may supply the estimatedposition information to the UAV via a communication link. In someembodiments, estimated position information may be calculated and sentto the UAV at periodic intervals. In particular, the UAV may calculateestimated position information of the UAV, which may be based on datafrom an on-board camera, and optionally at least one sensor and/or GPSreceiver, as described. When the user tracking mode is initiated, theUAV may perform an initial movement to track the target user based onthe received estimated position information and the current UAVposition. Such initial movement may establish a particular distanceand/or elevation of the UAV relative to the target user. During the usertracking mode, upon receiving estimated position information from thecontrol device, UAV may update/adjust its motion. For example, in someembodiments, the UAV motion may involve identifying changes in theposition of the control device and applying analogous changes to the UAVposition (e.g., comparing estimated position information to precedingestimated position information). In some embodiments, the UAV motion mayinvolve evaluating the position of the target user and maintaining thedistance and/or elevation of the UAV relative to the target user (e.g.,by comparing the estimated position information to a current UAVposition).

The various embodiments may enable the UAV in an automatic user trackingmode to receive an estimated position of a target user with improvedspeed and accuracy. In particular, a camera on the control device mayhave a frame rate of 30-60 Hz, and inertial sensors (e.g.,accelerometer, gyroscope, etc.) on the control device may have samplingrates of up to 500 Hz. Compared to estimated position information thatis determined, for example, based on GPS data alone (e.g., 1 Hz updaterate), the control device may calculate estimated position informationusing camera data, either alone or with other sensors, with a higheraccuracy, as well as enabling faster reaction times by the UAV.

In some embodiments, data from an imaging system and one or moreinertial sensor may be used to determine the position of the target userrelative to the surroundings. In particular, the control device positionmay be determined based on processing a sequence of images to derivecontrol device position information. Such approach may be based ontechniques for deriving “egomotion” (i.e., three-dimensional motion) ofa camera, which is often performed as part of “structure from motion”(SFM) methods. Specifically, SFM methods involve correlating a series ofimages taken from a moving camera, which are processed to simultaneouslyderive both a three dimensional model of the viewed scene and theegomotion of the camera.

In some embodiments, a hybrid approach employing both image processingand inertial sensor measurements may be used, either providing driftcancellation to the inertial sensors based on the image processing orproviding estimated motion parameters to the image processing system toimprove accuracy of calculations.

FIG. 2 illustrates the UAV 100 automatically tracking and following auser 200 based on information communicated by control device, such as awireless communication device 240 carried by the user. In the exampleillustrated in FIG. 2, the UAV 100 is being controlled to fly aparticular distance away and above the user 200 as the user 200 moves(e.g., walking, running, skiing, biking, etc.). Periodically, the user'swireless communication device 240 communicates the user's location tothe UAV 100 via wireless communications 260-266. Some of the userlocation updates communicated to the UAV 100 may be based on or includeGPS data obtained by a GPS receiver within the wireless communicationdevice 240. In addition, the user location updates may further includeinformation regarding motions and changes of direction of the user 200that are determined by the wireless communication device 240 based onimaging objects in the field of view 250 of a camera. For example, theuser 200 may be wearing the wireless communication device 240 on a beltso that a camera is able to image (e.g., video) the surroundings of userwithin the camera's field of view 250.

As the user 200 moves, objects in the background within the field ofview 250 shift positions. By determining the change in location withinthe field of view 250 of background objects from one image to the nextdivided by the time between each image (e.g., every thirtieth of asecond for a 30 frames-per-second video), a processor within thewireless communication device 240 is able to calculate movements of theuser 200. If the user 200 is moving in a straight line, the positionalshifts of background objects within the field of view 250 from frame toframe the used to calculate the instantaneous velocity of user,particularly if a distance to the background objects is known. If theuser is turning, background objects within the field of view 250 willappear to move quickly, depending upon the rate of turning or spending.

Because video frames are obtained approximately every 30^(th) of asecond, the information obtained by tracking changes in location ofobjects within the field of view 250 from one frame to the next cantrack movements of the user faster than may be possible based on GPSalone. Also, tracking changes in location of objects within the field ofview 250 from one frame to the next can detect changes in direction ofthe user 200, and thus may enable a processor (e.g., within the wirelesscommunication device 240 and/or the UAV 100) to predict the futuredirection of the user 200 faster that can be achieved using GPS and/oraccelerometer data. Thus, the various and embodiments may enable a moreresponsive tracking of the user 200 by the UAV 100 operating in thetracking mode.

Periodically, such as every few frames that are captured and analyzed,the wireless communication device 240 may transmit updated user positioninformation via wireless transmissions 260-266 to the UAV 100. Forexample, the wireless communication device 240 may transmit GPScoordinates of the user 200 to the UAV 100 at a first periodic rate,such as every 10 seconds (e.g., transmissions 260 and 266), while morefrequently transmitting user movement and direction updates based uponprocessing of background objects within images (e.g., transmissions 262and 264).

FIG. 3 is a functional block diagram of an example wirelesscommunication device 300 that is suitable for controlling a UAV duringan automatic user tracking mode in the various embodiments. According tovarious embodiments, the wireless device 300 may be the control device240 described with reference to FIG. 2.

With reference to FIGS. 1-3, the wireless communication device 300 maybe, for example, a cellular telephone, a personal digital assistant(PDA), a smartphone, a tablet, a wristband, an ankle band, a ring, awatch, a pendant, a belt, or any other type of portable or wearabledevice. In various embodiments, while in an automatic user trackingmode, the wireless communication device 300 may serve both functions ofcontrolling a UAV (e.g., 100) and providing estimated positions of thetarget user to the UAV.

The wireless communication device 300 may include at least one sensor302, such as one or more inertial sensor. For example, inertial sensorsmay include accelerometers (e.g., three-axis accelerometer),magnetometers (e.g., a 3-axis magnetometer), and/or gyroscopes (e.g., athree-axis gyroscope). The sensor(s) 302 may also include a barometer,which may be used to measure ambient pressure, and therefore toapproximate the elevation of the target user. The wireless communicationdevice 300 may also include a GPS unit 304 coupled to at least oneantenna 306 tuned to the GPS signaling channel. The GPS unit 304 maygenerate location signals corresponding to a location of the wirelesscommunication device 300 in response to GPS signals received thereby(i.e., when GPS signals are available).

The wireless communication device 300 may include an RF resource 308that is coupled to at least one antenna 309 and configured tocommunicate user position information to UAVs as described above. Invarious embodiments, the RF rousers 308 and GPS unit 304 may be coupledto a general-purpose processor 310. The RF resource 308 may includereceive circuitry for demodulating and decoding RF signals in order torecover operational parameter signals that are provided to thegeneral-purpose processor 310. The RF resource 308 may include transmitcircuitry for generating RF signals in response to operational controlsignals for transmission across the wireless link to the UAV. In variousembodiments, the RF resource 308 may implement separate transmit andreceive functionalities, or may include a transceiver that combinestransmitter and receiver functions.

In various embodiments, the RF resource 308 may wirelessly communicatewith a UAV via one or more wireless communication protocols, such asWi-Fi Bluetooth, or other long-range or short-range RF communication. Insome embodiments, the wireless communication device 300 may sendestimated position calculations based on sensor data to the UAV. In someembodiments, the wireless communication device 300 may additionally oralternatively transmit the raw data from the various sensors and/or GPSunit to the UAV.

In various embodiments, the wireless communication device 300 mayinclude a general-purpose processor 310, which may be a processing unit,application specific integrated circuit (ASIC), a field programmablegate array (FPGA), or other electronic circuitry for performingcomputations. The general-purpose processor 310 may be coupled to acoder/decoder (CODEC) 312. The CODEC 312 may in turn be coupled to aspeaker 314 and a microphone 316. The general-purpose processor may alsobe coupled to a memory 318 such as non-transitory computer readablestorage medium.

The memory 318 may store executable instructions to configure thegeneral-purpose processor 310 to implement the processes disclosed inthis disclosure. For example, the general-purpose processor 310 mayprocess the sensor readings of the sensors 302 and perform one or moresensor fusion algorithm to calculate estimated position information.

In various embodiments, the wireless communication device 300 includesan imaging system 320. The imaging system 320 may be coupled to thegeneral-purpose processor 310 and to one or more camera 322. The imagingsystem 320 may include image processing circuitry and a local storagefor image data (e.g., photographs and/or video) captured by the one ormore camera 322. To enable rapid analysis of images, the imaging system320 may make use of buffer memory that holds only a few frames forprocessing. In some embodiments, the local storage may be a separatestorage device, examples of which may include universal serial bus (USB)drives, memory cards, solid-state drives (SSDs), hard disk drives(HDDs), floppy disks, optical disks, magnetic tapes, and the like. Forexample, the local storage may be a removable storage device such as amemory card, for example, a PC Card, CompactFlash, SmartMedia, MemoryStick, Memory Stick Duo, Memory Stick PRO Duo, Miniature Card,Multimedia Card (MMC), Reduced Size Multimedia Card (RS-MMC), MMCmicroCard (MMCmicro), PS2 card, Secure Digital (SD) card, S×S, UniversalFlash Storage (UFS), miniSD, microSD, xD-Picture Card, Intelligent Stick(iStick), etc.

In some embodiments, the local storage may be a partition or part of thememory device 318. The image processing circuitry may be implemented bythe general-purpose processor 310, or may be implemented in a separateprocessing unit. The camera(s) 322 may include sub-components other thanimage capturing devices, including auto-focusing circuitry, ISOadjustment circuitry, and shutter speed adjustment circuitry, etc.

The imaging system 320 may perform various tasks including imagefiltering, image calling, video frame sampling, and other imageprocessing, audio processing, and/or video processing techniques. Inconjunction with the camera(s) 322, the imaging system 320 may beconfigured to capture pictures, videos, or any other image data with anysuitable parameters such as width, height, aspect ratio, megapixelcount, resolution or quality, etc. For example, the imaging device maybe configured to capture high-definition or ultra-high-definition videos(e.g., 720p, 1080i, 1080p, 1440p, 2000p, 2160p, 2540p, 4000p, 4320p,etc.).

Raw image data captured by the camera(s) 322 may be pre-processed toextract specific pieces of information. Examples of pre-processing mayinclude re-sampling to assure the correctness of the image coordinatesystem, noise reduction, contrast enhancement, scale spacerepresentation, etc. In some embodiments, the image processing circuitrymay further perform various image processing tasks (i.e., imageanalysis) on the raw data and/or pre-processed image data from thecamera(s) 322. In various embodiments, the image analysis may includeany of a number of example processing tasks. For example, the imageanalysis may include feature extraction at any suitable level ofcomplexity, image segmentation, data verification, image recognition,image registration, image matching, etc. The image analysis may beperformed using any of variety of methods that determine movement fromvisual data, referred to as “visual odometry.”

For example, the wireless communication device processor may calculatethe motion (e.g., speed of travel) and/or change in direction of thewireless communication device 300 over the time between images bydetermining the relative change in position of recognized features inthe field of view. Such features could be low-level computer visionfeatures detected using any of a number of techniques. For example, infeatures from accelerated segment test (FAST) corner detection, a circleof 16 pixels is used to classify whether a candidate center point isactually a corner. Specifically, if a set of contiguous pixels (e.g., 9pixels) in the circle are all brighter or darker than the center pixelintensity by at least a threshold value, the candidate point isclassified as a corner. Other corner detection methods that may be usedinclude, for example, Harris corner detection.

In some embodiments, features may be detected within image data usingalgorithms that are typically employed in object recognition tasks. Forexample, some embodiments may utilize scale-invariant feature transform(SIFT) and/or speeded up robust features (SURF) algorithms, in whichfeatures are compared to a database of features extracted from a set ofreference images.

In various embodiments, feature detection within image data may beimproved by selecting well-distributed features. For example, an imageor frame may be divided into a grid, and a number of features may beextracted from each section. Features identified in spaced apartsections may then be tracked from frame to frame for estimating motion,speed and direction.

In some embodiments feature tracking techniques may be employed, such asmulti-resolution (e.g., coarse-to-fine) tracking within image data.Feature tracking between images or frames may be improved in variousembodiments by estimating a surface normal in a manner that accounts forappearance transformation between views.

In some embodiments, data created by the imaging system 320 as a resultof the image analysis (i.e., processed image data) may be used by thegeneral-purpose processor 310 to calculate estimated positioninformation as described. In some embodiments, the general-purposeprocessor 310 may combine the processed image data with data collectedfrom other sensor(s) 304 and/or from the GPS unit 306 of the wirelesscommunication device 300 in order to calculate the estimated positioninformation. The set of the processed image data with, optionally, othersensor data and/or GPS data that may be used to calculate the estimatedposition information may be collectively referred to herein as “trackingdata.”

In some embodiments, the general purpose processor 310, memory 318, RFresource 308, GPS unit 304, and imaging system 320 may be included in asystem-on-chip device 333. The sensor(s) 302, the camera(s) 322, and theantennas 306, 309, as well as various input and output devices may becoupled to components of the system-on-chip device 333, such asinterfaces or controllers. Example user input components suitable foruse in the wireless communication device 300 may include, but are notlimited to, a keypad 324 and a touchscreen display 326. In someembodiments, the touchscreen display 326 may be used to implement a userinterface to control and monitor the UAV. In some embodiments, thetouchscreen display 326 may implement a graphical user interface (GUI)that presents visual information to a user of the wireless communicationdevice 300. The visual information displayed by the GUI may include agraphical map view depicting a location of the UAV, information aboutthe UAV's estimated position in relation to its environment, apredetermined movement path for the UAV once it exists the user trackingmode, and a display of signals received from the UAV, a visual feed viewgenerated from visual signals received from the UAV, etc.

FIGS. 4A-4C illustrate methods 400, 430, 450 for operating a wirelesscommunication device associated with a target user to support a UAV inan automatic user tracking mode, according to various embodiments. Invarious embodiments, the operations of the methods 400, 430, 450 may beperformed by the control device 240 in FIG. 2 and/or the wirelesscommunication device 300 in FIG. 3. In various embodiments, theoperations of the methods 400, 430, 450 may be implemented by one ormore processors associated with the wireless communication device. Suchone or more processors may include, for example, a processor of thewireless communication device, such as the general purpose processor 310in FIG. 3, or a separate controller (not shown) that may be coupled tothe memory 318, to the imaging system 320, and/or the sensor(s) 302 inFIG. 3. The operations of the methods 400, 430, 450 in variousembodiments involve calculating on the wireless communication deviceestimated position information (i.e., an estimated position, change inposition information, or combined position information), and providingthe position information to the UAV, optionally with GPS location data,for tracking a target user.

With reference to FIGS. 1-4A, the method 400 may be initiated while thewireless communication device is connected to a UAV (e.g., 100) andsupporting the UAV in the automatic user tracking mode. In someembodiments, the start of the automatic user tracking mode may betriggered in response to user input requesting activation of such mode,and/or by receiving signals from the UAV regarding its operating mode.In some embodiments, the automatic user tracking mode may be initiatedbased on the device satisfying various pre-programmedconditions/criteria. That is, in various embodiments, the wirelesscommunication device processor may be configured with sets of conditionsand/or criteria for launching a plurality of different tracking modes.In various embodiments, the wireless communication device may use the RFresource 308 to communicate with the UAV via one or more wirelesscommunication protocols. In some embodiments, the wireless communicationdevice may use the connection with the UAV to provide initial targetinformation to the UAV over the communication network. Such informationmay include the initial position, location, elevation, orientation,size, and/or other information about the wireless communication deviceand/or the target user. In some embodiments, the initial targetinformation may include an identification code, authentication key,International Mobile Equipment Identity (IMEI)/Electronic Serial Number(ESN) and/or other information that provides details about the targetuser and/or the wireless communication device.

In block 402, the wireless communication device processor may captureimage data of the wireless communication device's surroundings using atleast one camera associated with (i.e., on or connected/coupled to) thewireless communication device. In various embodiments, the capturedimage data may include a sequence of still images and/or a video streamto be analyzed for obtaining tracking data. As described, the image datamay be processed by an imaging system (e.g., 320) that is coupled to theat least one camera (e.g., 322) on the wireless communication device.

In block 404, the wireless communication device processor may analyzethe processed image data to calculate estimated position informationassociated with the target user. In some embodiments, calculatingestimated position information may involve calculating a change inposition information, which may be a change in the user's location,movement and/or direction of travel. For example, the wirelesscommunication device processor may calculate the motion (e.g., speed oftravel) and/or change in direction of the wireless communication deviceover the time between each image by determining the relative change inposition of recognized objects in the field of view and dividing by thetime between images. The image analysis may be performed using any ofvariety of visual odometry techniques as described. As described, suchcalculations may involve identifying a position of the wirelesscommunication device relative to the current environment. For example,using sequential image data captured by one or more camera (e.g. 322)over a period of time, the wireless communication device processor maycompute a distance traveled, a change in elevation, and/or a degree ofrotation by the device over the time between each captured image and/orvideo frame by tracking features or objects in the images. Any suitableimage recognition or identification techniques may be used to identifythe features/objects within the captured image data, includingapproaches based on CAD-like object models, appearance-based methods(e.g., using edge matching, grayscale matching, gradient matching,histograms of receptive field responses, or large model bases),feature-based methods (e.g., using interpretation trees, hypothesizingand testing, pose consistency, pose clustering, invariance, geometrichashing, Scale-invariant feature transform (SIFT), or Speeded Up RobustFeatures (SURF)), etc.

In block 406, the wireless communication device may transmit to the UAVthe estimated position information determined based on the analysis ofthe image data for the UAV's use in tracking the target user. In someembodiments, the estimated position information may be a calculatedchange in position, which may be used by the UAV, for example, to stayin the same relative position with respect to the wireless communicationdevice. In some embodiments, the estimated position information may be acurrent estimated position of the wireless communication device, whichmay be used by the UAV to calculate the wireless communication device'schange in position.

In determination block 408, the wireless communication device processormay determine whether the UAV is still operating in the automatic usertracking mode. For example, the wireless communication device mayreceive a user input requesting termination of the automatic usertracking mode, and may send an instruction indicating termination of theautomatic user tracking mode of operation by the UAV. Therefore, in someembodiments, the wireless communication device processor may perform thedetermination of whether the UAV is still operating in the automaticuser tracking mode without receiving a subsequent communication from theUAV. In some embodiments, the UAV may exit the automatic user trackingmode based on external conditions and/or pre-programmed parameters. Forexample, in some embodiments the UAV may have a default setting thatends the automatic user tracking mode after a particular period of timeunless otherwise instructed. In some embodiments, the UAV may beconfigured to terminate the automatic user tracking mode upon detectinga low power state, sustained signal loss, equipment failure, etc. Insome embodiments the wireless communication device may receive anotification that the automatic user tracking mode is terminated fromthe UAV over the established network link. In some embodiments, thewireless communication device may be configured to periodically transmitconfirmation requests/pings to the UAV, which may respond withacknowledgment signals confirming the continuation of the automatic usertracking mode.

In response to determining that the UAV is still operating in theautomatic user tracking mode (i.e., determination block 408=“Yes”), thewireless communication device processor may continue capturing imagedata of the wireless communication device surroundings in block 402.

In response to determining that the UAV is not still operating in theautomatic user tracking mode (i.e., determination block 408=“No”), thewireless communication device processor may end the method 400.

In some embodiments, the tracking data may also include data received bya GPS unit (e.g., 304) in addition to the image data. Referring to FIGS.1-4B, the method 430 may be initiated while the wireless communicationdevice is connected to a UAV (e.g., 100) and supporting the UAV in theautomatic user tracking mode. Similar to the method 400, in someembodiments, the start of the automatic user tracking mode may betriggered in response to a user input requesting such mode, by receivingsignals from a server or other computing device through a network link,and/or based on the device satisfying pre-programmedconditions/criteria. Also similar to method 400, in some embodiments thewireless communication device may use the connection with the UAV toprovide the initial target information to the UAV over the communicationnetwork.

In block 432, the wireless communication device processor may obtainupdated GPS location data from a GPS receiver associated with (i.e., onor coupled/connected to) the wireless communication device.

In block 402 (which may occur before, after, or simultaneously withblock 432), the wireless communication device processor may captureimage data of the wireless communication device's surroundings using atleast one camera associated with (i.e., on or connected/coupled to) thewireless communication device. In various embodiments, the capturedimage data may include a sequence of still images and/or a video streamof images to be analyzed to obtain tracking data. As described, theimage data may be processed by an imaging system (e.g., 320) that iscoupled to the at least one camera (e.g., 322) on the wirelesscommunication device.

In block 404, the wireless communication device processor may analyzethe processed image data to calculate estimated position informationassociated with the target user as described. For example, calculatingestimated position information may involve calculating a change inposition information (e.g., based on calculating motion and/or change indirection) associated with wireless communication device.

In block 434, the wireless communication device may transmit to the UAVthe updated GPS location data and the estimated position informationbased on the analyzed image data for the UAV's use in tracking thetarget user.

In determination block 408, the wireless communication device processormay determine whether the UAV is still operating in the automatic usertracking mode. As described, in some embodiments the wirelesscommunication device may receive a notification that the automatic usertracking mode is terminated from the UAV over the established networklink. In some embodiments, the wireless communication device may beconfigured to periodically transmit confirmation requests/pings to theUAV, which may respond with acknowledgment signals confirming thecontinuation of the automatic user tracking mode.

In response to determining that the UAV is still operating in theautomatic user tracking mode (i.e., determination block 408=“Yes”), thewireless communication device processor may continue obtaining updatedGPS data through a GPS receiver associated with (i.e., on orconnected/coupled to) the wireless communication device in block 432 andobtaining images in block 402. In some embodiments, the wirelesscommunication device processor may be configured to wait predeterminedtime duration before obtaining updated GPS data in block 432, althoughthe camera system may continue obtaining and processing images in blocks402 and 404. Such predetermined time duration, or the absence thereof,may be based on the types and capabilities of the various sensors thatprovide tracking data on the wireless communication device.

In response to determining that the UAV is not still operating in theautomatic user tracking mode (i.e., determination block 408=“No”), thewireless communication device processor may end the method 430.

In some embodiments, the tracking data may also include data captured byone or more sensor (e.g., 302), such as inertial sensor(s), a barometer,etc., in addition to the data received by the GPS unit (e.g., 304) andthe image data. Referring to FIGS. 1-4C, the method 450 may be initiatedwhile the wireless communication device is connected to a UAV (e.g.,100) and supporting the UAV in the automatic user tracking mode. Similarto methods 400, 430, in some embodiments, the start of the automaticuser tracking mode may be triggered in response to user input requestingsuch mode, by receiving signals from a server or other computing devicethrough a network link, and/or based on the device satisfyingpre-programmed conditions/criteria. Also similar to methods 400, 430, insome embodiments, the wireless communication device may use theconnection with the UAV to provide the initial target information to theUAV over the communication network.

In block 432, the wireless communication device processor may obtainupdated GPS location data through a GPS receiver on the wirelesscommunication device as described. In block 402, the wirelesscommunication device processor may capture image data of the wirelesscommunication device's surroundings using at least one camera associatedwith the wireless communication device as described. In block 404, thewireless communication device processor may analyze the processed imagedata to calculate estimated position information associated with thetarget user as described. For example, in some embodiments the wirelesscommunication device processor may calculate change in positioninformation (e.g., based on calculating motion and/or change indirection) associated with the wireless communication device asdescribed. In some embodiments, the wireless communication deviceprocessor may calculate a current estimated position of the wirelesscommunication device as described.

In block 452 (which may be performed before, after, or simultaneouslywith block 432 and/or block 402), the wireless communication deviceprocessor may obtain output data from one or more accelerometers and/ora gyroscope associated with (i.e., on or connected/coupled to) thewireless communication device. Such sensors provide information based ondirect movements of the wireless communication device.

In block 454, the wireless communication device processor may generatecombined position information based on the accelerometer and/orgyroscope data in combination with or consideration of the estimatedposition information (e.g., calculated change in position informationand/or current estimated position) obtained from processing images inblock 404. Information from accelerometers and/or gyroscopes may behelpful in analyzing the movement of background objects in obtainedimages because both sources of information are directly measuringmovements of the wireless communication device. For example, analyzingimages alone may provide confusing results when the user is turningquickly in circles (e.g., spinning), but such motion may be detected andthus understood based on data from a gyroscope. Continuing this example,if data from a gyroscope indicates that the user is spinning, the rapidmovement of objects in the background of images may be discounted ordisregarded when providing position updates to the UAV because theuser's position is not changing as rapidly as the images might suggest.As another example, accelerometer data may be used to confirm movementcalculations obtained from analyzing images, and vice versa.

Generating such combined position information may involve applying oneor more sensor fusion algorithms to the data that merges the calculatedchange in position information and the inertial sensor data (e.g.,accelerometer output, gyroscope output, etc.) in order to identify aposition of the wireless communication device relative to its currentenvironment in a more comprehensive manner than the using the image dataalone. For example, using information calculated based on a set ofsequential images of the surroundings captured by one or more camera(e.g. 322) over a period of time, as well as output data from anaccelerometer and/or gyroscope, a sensor fusion algorithm may provide anenhanced estimation of the distance traveled, the change in elevation,and/or the degree of rotation by the device over that time span.Further, the fusion algorithm may incorporate into the computationvarious measurements of speed, orientation, atmospheric pressure,coordinate-based location, etc. collected on the wireless communicationdevice depending on the configured sensors. In particular, fusionalgorithms may involve visual-inertial odometry, which uses data fromone or more inertial sensor (e.g., accelerometers, gyroscopes, etc.) anddata from, and/or calculation results based on, the processed image data(e.g., the calculated estimated position information in block 404).

In block 456, the wireless communication device may transmit to the UAVthe updated GPS location data and the combined position information forthe UAV's use in tracking the target user. In some embodiments, theautomatic user tracking mode may be a semi-autonomous mode, and thewireless communication device may be configured to send user-directedcommands to the UAV along with position information. For example, thewireless communication device may be configured to instruct the UAV tochange speed and/or direction by implementing a particular velocityand/or yaw rate. In this manner, the user of the wireless communicationdevice (i.e., target user) may be afforded some control over the UAVthrough flight commands, even while the wireless communication devicebeing automatically tracked.

In some embodiments, the UAV may be provided with additionalcapabilities that enable sensing obstacles (e.g., a stereo pair ofcameras). In such embodiments, when the automatic user tracking mode issemi-autonomous, the change in position data, the combined positioninformation, and/or the GPS data from the wireless communication devicemay be used by the UAV to correct and/or override commands for UAVflight (e.g., an instructed velocity and/or yaw rate) received directlyfrom the user of the wireless communication device (i.e., target user).For example, using the change in position UAV may avoid flying into anobstacle (e.g., tree, building, etc.) even if such maneuvers conflictwith instructions previously or simultaneously received from the targetuser.

In determination block 408, the wireless communication device processormay determine whether the UAV is still operating in the automatic usertracking mode. As described in method 400, in some embodiments thewireless communication device may receive a notification that theautomatic user tracking mode is terminated from the UAV over theestablished network link. In some embodiments, the wirelesscommunication device may be configured to periodically transmitconfirmation requests/pings to the UAV, which may respond withacknowledgment signals confirming the continuation of the automatic usertracking mode.

In response to determining that the UAV is still operating in theautomatic user tracking mode (i.e., determination block 408=“Yes”), thewireless communication device processor may again obtain updated GPSlocation data from the GPS receiver on the wireless communication devicein block 432 and repeat the operations of the method 450.

In response to determining that the UAV is no longer operating in theautomatic user tracking mode (i.e., determination block 408=“No”), thewireless communication device processor may end the method 450.

FIG. 5 illustrates a method 500 for implementing an automatic usertracking mode on a UAV to track a wireless communication deviceassociated with a target user, according to various embodiments. Withreference to FIGS. 1-5, the UAV (e.g., 100) may be configured tocommunicate with at least one control device (e.g., 240), which may be awireless communication device (e.g., 300). In various embodiments, theoperations of the method 500 may be implemented by one or moreprocessors (e.g., 120) of the UAV, such as a general purpose processoror a separate controller (not shown) that may be coupled to the memory(e.g., 121), flight and/or position controllers, and other componentsand systems.

In block 502, the UAV processor may detect that the automatic usertracking mode has been activated on the UAV. In some embodiments, theUAV may receive from the control device instructions to initiate theautomatic user tracking mode.

In block 504, the UAV processor may receive initial target informationfrom the control device. As described, in various embodiments, theinitial target information may be any information that may be used toidentify the control device, and to identify the initial location of thecontrol device and/or target user.

In block 506, the UAV processor may calculate a current UAV position. Invarious embodiments, similar to the estimated position informationcalculated in method 400 (e.g., 408), the current UAV position may be aposition of the UAV relative to its surroundings. In some embodiments,the current UAV position may be the UAV's elevation and/or location. Invarious embodiments, the current UAV position may be calculated usingdata obtained from one or more on-board sensors of the UAV. For example,the current UAV position may be calculated by receiving, for example,image data captured by a camera, a barometer reading, a GPS z-axisreading, an inertial sensor reading, or any combination thereof. Invarious embodiments, the inertial sensors on the UAV may include one ormore of 3-axis accelerometers, 3-axis magnetometers, and 3-axisgyroscopes. In some embodiments, the current UAV position may bedetermined based on visual odometry solutions using image data capturedby a camera, as described with respect to the control device.

In various embodiments, the UAV may be configured with a separatecontroller or logic that tracks orientation of the UAV, stabilizing theUAV at a desired tilt angle. In various embodiments, the UAV may beconfigured with a separate controller or logic to keep a desiredsetpoint for the UAV position.

In block 508, the UAV may begin controlling flight by tracking thetarget user associated with the control device. The flight controls maybe based on the calculated current UAV position and the initial targetinformation. This process may involve flying to a position at a presetelevation and/or ground distance (“selected offset”) from the controldevice.

In block 510, the UAV processor may receive estimated positioninformation from the control device.

In determination block 512, the UAV processor may determine whether anupdate to the UAV motion is required. In various embodiments, updatingthe motion of the UAV device may involve, for example, adjusting the UAVvelocity, elevation, orientation, etc. of the UAV. In some embodiments,determining whether an update to the UAV motion is required may be basedon comparing received estimated position information from the controldevice to previous estimated position information, and detecting whetherthere is any difference. In some embodiments, determining whether anupdate to the UAV motion is required may involve comparing the estimatedposition information to the current UAV position, and detecting whethera preset elevation and/or ground distance (i.e., a selected offset) fromthe control device is maintained. Therefore, some embodiments mayrequire re-calculating the current UAV position upon receiving theestimated position information. For example, if the target user hasmoved forward five meters, the desired setpoint for the UAV position maybe adjusted five meters.

In response to determining that an update to the UAV motion is required(i.e., determination block 512=“Yes”), the UAV processor may adjust theUAV motion in block 514. For example, if a difference was detectedbetween the estimated position information and preceding estimatedposition information, the adjustment in the UAV motion may be acorresponding change in position of the UAV. In another example, if aselected offset from the control device was not maintained, the UAVmotion may be adjusted to return to that preset elevation and/ordistance. In various embodiments, the UAV processor may provideinstructions that cause the thrust power of at least one propellerdriver to accomplish the required motion adjustment.

Following block 514 or in response to determining that an update to theUAV motion is not required (i.e., determination block 512=“No”), the UAVprocessor may determine whether to exit the user tracking mode indetermination block 516. As described, the automatic user tracking modemay be stopped, for example, as a result of user input to the controldevice, or based on external conditions and/or preset parameters thatensure the quality of the UAV performance.

In response to determining not to exit the automatic user tracking mode(i.e., determination block 516=“No”), the UAV processor may return toblock 510 to receive estimated position information from the controldevice. In response to determining to exit the automatic user trackingmode (i.e., determination block 516=“Yes”), the UAV processor may endthe method 500 and transition to a different type of flight control.

The various embodiments may be implemented within a variety of UAVs, anexample of which in the form of a four-rotor UAV is illustrated in FIG.6 that is suitable for use with the various embodiments including theembodiments described with reference to FIG. 5. With reference to FIGS.1-6, the UAV 100 may include a body 600 (i.e., fuselage, frame, etc.)that may be made out of any combination of plastic, metal, or othermaterials suitable for flight. The body 600 may include a processor 630that is configured to monitor and control the various functionalities,subsystems, and/or other components of the UAV 100. For example, theprocessor 630 may be configured to monitor and control variousfunctionalities of the UAV 100, such as any combination of modules,software, instructions, circuitry, hardware, etc. related to propulsion,navigation, power management, sensor management, and/or stabilitymanagement.

The processor 630 may include one or more processing unit(s) 601, suchas one or more processors configured to execute processor-executableinstructions (e.g., applications, routines, scripts, instruction sets,etc.), a memory and/or storage unit 602 configured to store data (e.g.,flight plans, obtained sensor data, received messages, applications,etc.), and a wireless transceiver 604 and antenna 606 for transmittingand receiving wireless signals (e.g., a Wi-Fi® radio and antenna,Bluetooth®, RF, etc.). In some embodiments, the UAV 100 may also includecomponents for communicating via various wide area networks, such ascellular network transceivers or chips and associated antenna (notshown). In some embodiments, the processor 630 of the UAV 100 mayfurther include various input units 608 for receiving data from humanoperators and/or for collecting data indicating various conditionsrelevant to the UAV 100. For example, the input units 608 may includecamera(s), microphone(s), location information functionalities (e.g., aglobal positioning system (GPS) receiver for receiving GPS coordinates),flight instruments (e.g., attitude indicator(s), gyroscope(s),accelerometer(s), altimeter(s), compass(es), etc.), keypad(s), etc. Thevarious components of the processor 630 may be connected via a bus 610or other similar circuitry.

The body 600 may include landing gear 620 of various designs andpurposes, such as legs, skis, wheels, pontoons, etc. The body 600 mayalso include a payload mechanism 621 configured to hold, hook, grasp,envelope, and otherwise carry various payloads, such as boxes. In someembodiments, the payload mechanism 621 may include and/or be coupled toactuators, tracks, rails, ballasts, motors, and other components foradjusting the position and/or orientation of the payloads being carriedby the UAV 100. For example, the payload mechanism 621 may include a boxmoveably attached to a rail such that payloads within the box may bemoved back and forth along the rail. The payload mechanism 621 may becoupled to the processor 630 and thus may be configured to receiveconfiguration or adjustment instructions. For example, the payloadmechanism 621 may be configured to engage a motor to re-position apayload based on instructions received from the processor 630.

The UAV 100 may be of a helicopter design that utilizes one or morerotors 624 driven by corresponding motors 622 to provide lift-off (ortake-off) as well as other aerial movements (e.g., forward progression,ascension, descending, lateral movements, tilting, rotating, etc.). TheUAV 100 may utilize various motors 622 and corresponding rotors 624 forlifting off and providing aerial propulsion. For example, the UAV 100may be a “quad-copter” that is equipped with four motors 622 andcorresponding rotors 624. The motors 622 may be coupled to the processor630 and thus may be configured to receive operating instructions orsignals from the processor 630. For example, the motors 622 may beconfigured to increase rotation speed of their corresponding rotors 624,etc. based on instructions received from the processor 630. In someembodiments, the motors 622 may be independently controlled by theprocessor 630 such that some rotors 624 may be engaged at differentspeeds, using different amounts of power, and/or providing differentlevels of output for moving the UAV 100. For example, motors 622 on oneside of the body 600 may be configured to cause their correspondingrotors 624 to spin at a higher rotations per minute (RPM) than rotors624 on the opposite side of the body 600 in order to balance the UAV 100burdened with an off-centered payload.

The body 600 may include a power source 612 that may be coupled to andconfigured to power the various other components of the UAV 100. Forexample, the power source 612 may be a rechargeable battery forproviding power to operate the motors 622, the payload mechanism 621,and/or the units of the processor 630.

The various processors described herein may be any programmablemicroprocessor, microcomputer or multiple processor chip or chips thatcan be configured by software instructions (applications) to perform avariety of functions, including the functions of the various embodimentsdescribed herein. In the various devices, multiple processors may beprovided, such as one processor dedicated to wireless communicationfunctions and one processor dedicated to running other applications.Typically, software applications may be stored in internal memory beforethey are accessed and loaded into the processors. The processors mayinclude internal memory sufficient to store the application softwareinstructions. In many devices, the internal memory may be a volatile ornonvolatile memory, such as flash memory, or a mixture of both. For thepurposes of this description, a general reference to memory refers tomemory accessible by the processors including internal memory orremovable memory plugged into the various devices and memory within theprocessors.

In various embodiments, the control device that may control and betracked by UAV 100 through cellular networks, or other communicationlinks, may be any of a variety of wireless communication devices (e.g.,smartphones, tablets, etc.) an example in the form of a smartphone orwireless communication device 700 is illustrated in FIG. 7. Withreference to FIGS. 1-7, the wireless communication device 700 mayinclude a processor 702 coupled to the various systems of the wirelesscommunication device 700. For example, the processor 702 may be coupledto a touch screen controller 704, radio communication elements, speakersand microphones, and an internal memory 706. The processor 702 may beone or more multi-core integrated circuits designated for general orspecific processing tasks. The internal memory 706 may be volatile ornon-volatile memory, and may also be secure and/or encrypted memory, orunsecure and/or unencrypted memory, or any combination thereof. Inanother embodiment (not shown), the wireless communication device 700may also be coupled to an external memory, such as an external harddrive.

The touch screen controller 704 and the processor 702 may also becoupled to a touch screen panel 712, such as a resistive-sensing touchscreen, capacitive-sensing touch screen, infrared sensing touch screen,etc. Additionally, the display of the wireless communication device 700need not have touch screen capability. The wireless communication device700 may have one or more radio signal transceivers 708 (e.g., Peanut,Bluetooth, Bluetooth LE, Zigbee, Wi-Fi, RF radio, etc.) and antennae710, for sending and receiving communications, coupled to each otherand/or to the processor 702. The transceivers 708 and antennae 710 maybe used with the above-mentioned circuitry to implement the variouswireless transmission protocol stacks and interfaces. The wirelesscommunication device 700 may include a cellular network wireless modemchip 716 that enables communication via a cellular network and iscoupled to the processor.

The wireless communication device 700 may include a peripheral deviceconnection interface 718 coupled to the processor 702. The peripheraldevice connection interface 718 may be singularly configured to acceptone type of connection, or may be configured to accept various types ofphysical and communication connections, common or proprietary, such asUSB, FireWire, Thunderbolt, or PCIe. The peripheral device connectioninterface 718 may also be coupled to a similarly configured peripheraldevice connection port (not shown).

In some embodiments, the wireless communication device 700 may includemicrophones 715 a-715 c. For example, the wireless communication devicemay have a conventional microphone 715 a for receiving voice or otheraudio frequency energy from a user during a call. The wirelesscommunication device 700 may further be configured with additionalmicrophones 715 b and 715 c, which may be configured to receive audioincluding ultrasound signals. Alternatively, all microphones 715 a, 715b, and 715 c may be configured to receive ultrasound signals. Themicrophones 715 may be piezo-electric transducers, or other conventionalmicrophone elements. Because more than one microphone 715 may be used,relative location information may be received in connection with areceived ultrasound signal through various triangulation methods. Atleast two microphones 715 configured to receive ultrasound signals maybe used to generate position information for an emitter of ultrasoundenergy.

The wireless communication device 700 may also include speakers 714 forproviding audio outputs. The wireless communication device 700 may alsoinclude a housing 720, constructed of a plastic, metal, or a combinationof materials, for containing all or some of the components discussedherein. The wireless communication device 700 may include a power source722 coupled to the processor 702, such as a disposable or rechargeablebattery. The rechargeable battery may also be coupled to the peripheraldevice connection port to receive a charging current from a sourceexternal to the wireless communication device 700. The wirelesscommunication device 700 may also include a physical button 724 forreceiving user inputs. The wireless communication device 700 may alsoinclude a power button 726 for turning the wireless communication device700 on and off.

In some embodiments, the wireless communication device 700 may furtherinclude an accelerometer 728, which senses movement, vibration, andother aspects of the device through the ability to detectmulti-directional values of and changes in acceleration. In variousembodiments, the accelerometer 728 may be used to determine the x, y,and z positions of the wireless communication device 700. Using theinformation from the accelerometer, a pointing direction of the wirelesscommunication device 700 may be detected.

The processors 630, 702 may be any programmable microprocessor,microcomputer or multiple processor chip or chips that can be configuredby software instructions (applications) to perform a variety offunctions, including the functions of various embodiments describedabove. In some mobile devices, multiple processors may be provided, suchas one processor dedicated to wireless communication functions and oneprocessor dedicated to running other applications. Typically, softwareapplications may be stored in the internal memory 602, 706 before theyare accessed and loaded into the processors 630, 702. The processors630, 702 may include internal memory sufficient to store the applicationsoftware instructions. In many mobile devices the internal memory may bea volatile or nonvolatile memory, such as flash memory, or a mixture ofboth. For the purposes of this description, a general reference tomemory refers to memory accessible by the processors 630, 702 includinginternal memory or removable memory plugged into the mobile device andmemory within the processor processors 630, 702 themselves.

The various embodiments illustrated and described are provided merely asexamples to illustrate various features of the claims. However, featuresshown and described with respect to any given embodiment are notnecessarily limited to the associated embodiment and may be used orcombined with other embodiments that are shown and described. Further,the claims are not intended to be limited by any one example embodiment.

The foregoing method descriptions and the process flow diagrams areprovided merely as illustrative examples and are not intended to requireor imply that the steps of the various embodiments must be performed inthe order presented. As will be appreciated by one of skill in the artthe order of steps in the foregoing embodiments may be performed in anyorder. Words such as “thereafter,” “then,” “next,” etc. are not intendedto limit the order of the steps; these words are simply used to guidethe reader through the description of the methods. Further, anyreference to claim elements in the singular, for example, using thearticles “a,” “an” or “the” is not to be construed as limiting theelement to the singular.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and steps have been described generally in terms offunctionality. Whether such functionality is implemented as hardware orsoftware depends upon the particular application and design constraintsimposed on the overall system. Skilled artisans may implement thedescribed functionality in varying ways for each particular application,but such implementation decisions should not be interpreted as causing adeparture from the scope of the present claims.

The hardware used to implement the various illustrative logics, logicalblocks, modules, and circuits described in connection with the aspectsdisclosed herein may be implemented or performed with a general purposeprocessor, a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general-purpose processor maybe a microprocessor, but, in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of receiver smartobjects, e.g., a combination of a DSP and a microprocessor, a pluralityof microprocessors, one or more microprocessors in conjunction with aDSP core, or any other such configuration. Alternatively, some steps ormethods may be performed by circuitry that is specific to a givenfunction.

In one or more exemplary aspects, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored as one or moreinstructions or code on a non-transitory computer-readable storagemedium or non-transitory processor-readable storage medium. The steps ofa method or algorithm disclosed herein may be embodied inprocessor-executable software, which may reside on a non-transitorycomputer-readable or processor-readable storage medium. Non-transitorycomputer-readable or processor-readable storage media may be any storagemedia that may be accessed by a computer or a processor. By way ofexample but not limitation, such non-transitory computer-readable orprocessor-readable storage media may include RAM, ROM, EEPROM, FLASHmemory, CD-ROM or other optical disk storage, magnetic disk storage orother magnetic storage smart objects, or any other medium that may beused to store desired program code in the form of instructions or datastructures and that may be accessed by a computer. Disk and disc, asused herein, includes compact disc (CD), laser disc, optical disc,digital versatile disc (DVD), floppy disk, and Blu-ray disc where disksusually reproduce data magnetically, while discs reproduce dataoptically with lasers. Combinations of the above are also includedwithin the scope of non-transitory computer-readable andprocessor-readable media. Additionally, the operations of a method oralgorithm may reside as one or any combination or set of codes and/orinstructions on a non-transitory processor-readable storage mediumand/or computer-readable storage medium, which may be incorporated intoa computer program product.

The preceding description of the disclosed embodiments is provided toenable any person skilled in the art to make or use the claims. Variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to some embodiments without departing from the scope of theclaims. Thus, the claims are not intended to be limited to theembodiments shown herein but are to be accorded the widest scopeconsistent with the language of the claims and the principles and novelfeatures disclosed herein.

What is claimed is:
 1. A method performed by a processor associated witha wireless communication device for enabling an unmanned autonomousvehicle (UAV) to operate in an automatic user tracking mode, the methodcomprising: capturing image data of surroundings of the wirelesscommunication device by at least one camera associated with the wirelesscommunication device while the UAV is operating in the automatic usertracking mode; calculating estimated position information for thewireless communication device based on captured image data; andtransmitting the estimated position information to the UAV for use intracking a target user.
 2. The method of claim 1, further comprisingdetecting that the automatic user tracking mode is initiated based ondetecting user inputs on the wireless communication device.
 3. Themethod of claim 1, further comprising: determining whether the UAV isstill operating in the automatic user tracking mode; and repeatingcollecting image data, calculating estimated position information, andtransmitting the estimated position information to the UAV in responseto determining that the UAV is still operating in the automatic usertracking mode.
 4. The method of claim 1, wherein calculating estimatedposition information comprises calculating a change in position of thewireless communication device; and wherein calculating the change inposition comprises analyzing a sequence of the captured image data overa period of time to calculate a rate of movement of at least one featurein the surroundings.
 5. The method of claim 1, further comprising:obtaining updated location data received through a Global PositioningSystem (GPS) receiver associated with the wireless communication device;and transmitting the updated location data to the UAV for use intracking the target user.
 6. The method of claim 1, further comprising:obtaining inertial sensor output data from at least one of anaccelerometer, a gyroscope, and a magnetometer associated with thewireless communication device; and generating combined positioninformation for the wireless communication device based on the estimatedposition information and the inertial sensor output data; whereintransmitting the estimated position information to the UAV comprisestransmitting the combined position information.
 7. The method of claim1, further comprising transmitting: initial target information to theUAV, wherein the initial target information provides to the UAV aninitial location of the wireless communication device.
 8. The method ofclaim 7, wherein the initial target information provides to the UAVidentity data for at least one of the wireless communication device andthe target user.
 9. The method of claim 1, wherein the estimatedposition information transmitted to the UAV is configured to enable theUAV to determine whether an update to UAV motion is required.
 10. Themethod of claim 1, wherein calculating estimated position informationcomprises calculating a current estimated position of the wirelesscommunication device.
 11. A method performed by a processor of anunmanned autonomous vehicle (UAV) for enabling the UAV to automaticallyfollow a user, the method comprising: calculating a current position ofthe UAV; receiving estimated position information from a wirelesscommunication device associated with the user, wherein the estimatedposition information is derived from image data of surroundings of theuser captured by at least one camera of the wireless communicationdevice; and determining whether an update to UAV motion is required. 12.The method of claim 11, wherein receiving the estimated positioninformation from the wireless communication device comprises receiving achange in position calculated by the wireless communication device. 13.The method of claim 11, wherein receiving the estimated positioninformation from the wireless communication device comprises receiving acurrent estimated position of the wireless communication device, themethod further comprising: calculating a change in position of thewireless communication device using the received current estimatedposition.
 14. The method of claim 11, wherein determining whether anupdate to the UAV motion comprises: comparing the estimated positioninformation to previous position information received from the wirelesscommunication device; and detecting movement of the wirelesscommunication device based on the comparison.
 15. The method of claim11, wherein determining whether an update to the UAV motion comprises:comparing the estimated position information to a current UAV position;and determining whether the UAV has maintained a selected offset fromthe wireless communication device, wherein the selected offset comprisesa preset elevation or ground distance.
 16. A wireless communicationdevice, comprising: a transceiver configured to communicate with anunmanned autonomous vehicle (UAV); and a processor coupled to thetransceiver and configured to: obtain image data of surroundingscaptured by a camera associated with the wireless communication devicewhile the UAV is operating in an automatic user tracking mode; calculateestimated position information for the wireless communication devicebased on captured image data; and transmit the estimated positioninformation via the transceiver to the UAV for use in tracking a targetuser.
 17. The wireless communication device of claim 16, wherein theprocessor is further configured to detect that the automatic usertracking mode is initiated based on detecting user inputs on thewireless communication device.
 18. The wireless communication device ofclaim 16, wherein the processor is further configured to: determinewhether the UAV is still operating in the automatic user tracking mode;and repeat collecting image data, calculating estimated positioninformation, and transmitting the estimated position information to theUAV in response to determining that the UAV is still operating in theautomatic user tracking mode.
 19. The wireless communication device ofclaim 16, wherein the processor is further configured to: calculate theestimated position information by calculating a change in position ofthe wireless communication device, and calculate the change in positionby analyzing a sequence of the captured image data over a period of timeto calculate a rate of movement of at least one feature in thesurroundings.
 20. The wireless communication device of claim 16, whereinthe processor is further configured to: obtain updated location datareceived through a Global Positioning System (GPS) receiver associatedwith the wireless communication device; and transmit the updatedlocation data to the UAV for use in tracking the target user.
 21. Thewireless communication device of claim 16, wherein the processor isfurther configured to: obtain inertial sensor output data from at leastone of an accelerometer, a gyroscope, and a magnetometer associated withthe wireless communication device; and generate combined positioninformation for the wireless communication device based on the estimatedposition information and the inertial sensor output data, wherein theprocessor is further configured to transmit the combined positioninformation as the estimated position information transmitted to theUAV.
 22. The wireless communication device of claim 16, wherein theprocessor is further configured to transmit initial target informationto the UAV, wherein the initial target information provides to the UAVan initial location of the wireless communication device.
 23. Thewireless communication device of claim 22, wherein the initial targetinformation provides to the UAV identity data for at least one of thewireless communication device and the target user.
 24. The wirelesscommunication device of claim 16, wherein the estimated positioninformation transmitted to the UAV is configured to enable the UAV todetermine whether an update to UAV motion is required.
 25. The wirelesscommunication device of claim 16, wherein the processor is furtherconfigured to calculate the estimated position information bycalculating a current estimated position of the wireless communicationdevice.
 26. An unmanned autonomous vehicle (UAV), comprising: atransceiver; and a processor coupled to the transceiver and configuredto: calculate a current position of the UAV; receive estimated positioninformation from a wireless communication device associated with a user,wherein the estimated position information is derived from image data ofsurroundings of the user captured by at least one camera of the wirelesscommunication device; and determine whether an update to UAV motion isrequired.
 27. The UAV of claim 26, wherein the processor is furtherconfigured to receive a change in position calculated by the wirelesscommunication device as at least part of the estimated positioninformation from the wireless communication device.
 28. The UAV of claim26, wherein the processor is further configured to: receive a currentestimated position of the wireless communication device as at least partof the estimated position information from the wireless communicationdevice comprises receiving; and calculating a change in position of thewireless communication device using the received current estimatedposition.
 29. The UAV of claim 26, wherein the processor is furtherconfigured to determine whether an update to the UAV motion by:comparing the estimated position information to previous positioninformation received from the wireless communication device; anddetecting movement of the wireless communication device based on thecomparison.
 30. The UAV of claim 26, wherein the processor is furtherconfigured to determine whether an update to the UAV motion by:comparing the estimated position information to a current UAV position;and determining whether the UAV has maintained a selected offset fromthe wireless communication device, wherein the selected offset comprisesa preset elevation or ground distance.